Technology Aware Training in Memristive Neuromorphic Systems based on non-ideal Synaptic Crossbars

The advances in the field of machine learning using neuromorphic systems have paved the pathway for extensive research on possibilities of hardware implementations of neural networks. Various memristive technologies such as oxide-based devices, spintronics and phase change materials have been explored to implement the core functional units of neuromorphic systems, namely the synaptic network, and the neuronal functionality, in a fast and energy efficient manner. However, various non-idealities in the crossbar implementations of the synaptic arrays can significantly degrade performance of neural networks and hence, impose restrictions on feasible crossbar sizes. In this work, we build mathematical models of various non-idealities that occur in crossbar implementations such as source resistance, neuron resistance and chip-to-chip device variations and analyze their impact on the classification accuracy of a fully connected network (FCN) and convolutional neural network (CNN) trained with standard training algorithm. We show that a network trained under ideal conditions can suffer accuracy degradation as large as 59.84% for FCNs and 62.4% for CNNs when implemented on non-ideal crossbars for relevant non-ideality ranges. This severely constrains the sizes for crossbars. As a solution, we propose a technology aware training algorithm which incorporates the mathematical models of the non-idealities in the standard training algorithm. We demonstrate that our proposed methodology achieves significant recovery of testing accuracy within 1.9% of the ideal accuracy for FCNs and 1.5% for CNNs. We further show that our proposed training algorithm can potentially allow the use of significantly larger crossbar arrays of sizes 784$\times$500 for FCNs and 4096$\times$512 for CNNs with a minor or no trade-off in accuracyComment: 10 pages, 7 figures, submitted to IEEE Transactions on Emerging Topics in Computational Intelligenc

Tree-CNN: A Hierarchical Deep Convolutional Neural Network for Incremental Learning

Over the past decade, Deep Convolutional Neural Networks (DCNNs) have shown remarkable performance in most computer vision tasks. These tasks traditionally use a fixed dataset, and the model, once trained, is deployed as is. Adding new information to such a model presents a challenge due to complex training issues, such as "catastrophic forgetting", and sensitivity to hyper-parameter tuning. However, in this modern world, data is constantly evolving, and our deep learning models are required to adapt to these changes. In this paper, we propose an adaptive hierarchical network structure composed of DCNNs that can grow and learn as new data becomes available. The network grows in a tree-like fashion to accommodate new classes of data, while preserving the ability to distinguish the previously trained classes. The network organizes the incrementally available data into feature-driven super-classes and improves upon existing hierarchical CNN models by adding the capability of self-growth. The proposed hierarchical model, when compared against fine-tuning a deep network, achieves significant reduction of training effort, while maintaining competitive accuracy on CIFAR-10 and CIFAR-100.Comment: 8 pages, 6 figures, 7 tables Accepted in Neural Networks, 201

Exploring Ultra Low-Power on-Chip Clocking Using Functionality Enhanced Spin-Torque Switches

Emerging spin-torque (ST) phenomena may lead to ultra-low-voltage, high-speed nano-magnetic switches. Such current-based-switches can be attractive for designing low swing global-interconnects, like, clocking-networks and databuses. In this work we present the basic idea of using such ST-switches for low-power on-chip clocking. For clockingnetworks, Spin-Hall-Effect (SHE) can be used to produce an assist-field for fast ST-switching using global-mesh-clock with less than 100mV swing. The ST-switch acts as a compact-latch, written by ultra-low-voltage input-pulses. The data is read using a high-resistance tunnel-junction. The clock-driven SHE write-assist can be shared among large number of ST-latches, thereby reducing the load-capacitance for clock-distribution. The SHE assist can be activated by a low-swing clock (~150mV) and hence can facilitate ultra-low voltage clock-distribution. Owing to reduced clock-load and low-voltage operation, the proposed scheme can achieve 97% low-power for on-chip clocking as compared to the state of the art CMOS design. Rigorous device-circuit simulations and system-level modelling for the proposed scheme will be addressed in future

Image Edge Detection based on Swarm Intelligence using Memristive Networks

Recent advancements in the development of memristive devices has opened new opportunities for hardware implementation of non-Boolean computing. To this end, the suitability of memristive devices for swarm intelligence algorithms has enabled researchers to solve a maze in hardware. In this paper, we utilize swarm intelligence of memristive networks to perform image edge detection. First, we propose a hardware-friendly algorithm for image edge detection based on ant colony. Second, we implement the image edge detection algorithm using memristive networks. Furthermore, we explain the impact of various parameters of the memristors on the efficacy of the implementation. Our results show 28% improvement in the energy compared to a low power CMOS hardware implementation based on stochastic circuits. Furthermore, our design occupies up to 5x less area.Comment: This paper was submitted to IEEE Transactions on Computer Aided desig

Short-Term Plasticity and Long-Term Potentiation in Magnetic Tunnel Junctions: Towards Volatile Synapses

Synaptic memory is considered to be the main element responsible for learning and cognition in humans. Although traditionally non-volatile long-term plasticity changes have been implemented in nanoelectronic synapses for neuromorphic applications, recent studies in neuroscience have revealed that biological synapses undergo meta-stable volatile strengthening followed by a long-term strengthening provided that the frequency of the input stimulus is sufficiently high. Such "memory strengthening" and "memory decay" functionalities can potentially lead to adaptive neuromorphic architectures. In this paper, we demonstrate the close resemblance of the magnetization dynamics of a Magnetic Tunnel Junction (MTJ) to short-term plasticity and long-term potentiation observed in biological synapses. We illustrate that, in addition to the magnitude and duration of the input stimulus, frequency of the stimulus plays a critical role in determining long-term potentiation of the MTJ. Such MTJ synaptic memory arrays can be utilized to create compact, ultra-fast and low power intelligent neural systems.Comment: The article will appear in a future issue of Physical Review Applie

Encoding Neural and Synaptic Functionalities in Electron Spin: A Pathway to Efficient Neuromorphic Computing

Present day computers expend orders of magnitude more computational resources to perform various cognitive and perception related tasks that humans routinely perform everyday. This has recently resulted in a seismic shift in the field of computation where research efforts are being directed to develop a neurocomputer that attempts to mimic the human brain by nanoelectronic components and thereby harness its efficiency in recognition problems. Bridging the gap between neuroscience and nanoelectronics, this paper attempts to provide a review of the recent developments in the field of spintronic device based neuromorphic computing. Description of various spin-transfer torque mechanisms that can be potentially utilized for realizing device structures mimicking neural and synaptic functionalities is provided. A cross-layer perspective extending from the device to the circuit and system level is presented to envision the design of an All-Spin neuromorphic processor enabled with on-chip learning functionalities. Device-circuit-algorithm co-simulation framework calibrated to experimental results suggest that such All-Spin neuromorphic systems can potentially achieve almost two orders of magnitude energy improvement in comparison to state-of-the-art CMOS implementations.Comment: The paper will appear in a future issue of Applied Physics Review

Chaos-guided Input Structuring for Improved Learning in Recurrent Neural Networks

Anatomical studies demonstrate that brain reformats input information to generate reliable responses for performing computations. However, it remains unclear how neural circuits encode complex spatio-temporal patterns. We show that neural dynamics are strongly influenced by the phase alignment between the input and the spontaneous chaotic activity. Input structuring along the dominant chaotic projections causes the chaotic trajectories to become stable channels (or attractors), hence, improving the computational capability of a recurrent network. Using mean field analysis, we derive the impact of input structuring on the overall stability of attractors formed. Our results indicate that input alignment determines the extent of intrinsic noise suppression and hence, alters the attractor state stability, thereby controlling the network's inference ability.Comment: 11 pages with 5 figures including supplementary materia

Magnetic Skyrmions for Cache Memory

Magnetic skyrmions (MS) are particle-like spin structures with whirling configuration, which are promising candidates for spin-based memory. MS contains alluring features including remarkably high stability, ultra low driving current density, and compact size. Due to their higher stability and lower drive current requirement for movement, skyrmions have great potential in energy efficient spintronic device applications. We propose a skyrmion-based cache memory where data can be stored in a long nanotrack as multiple bits. Write operation (formation of skyrmion) can be achieved by injecting spin polarized current in a magnetic nanotrack and subsequently shifting the MS in either direction along the nanotrack using charge current through a spin-Hall metal (SHM), underneath the magnetic layer. The presence of skyrmion can alter the resistance of a magnetic tunneling junction (MTJ) at the read port. Considering the read and write latency along the long nanotrack cache memory, a strategy of multiple read and write operations is discussed. Besides, the size of a skyrmion affects the packing density, current induced motion velocity, and readability (change of resistance while sensing the existence of a skyrmion). Design optimization to mitigate the above effects is also investigated

Quantum theory of light emission from quantum dots coupled to structured photonic reservoirs and acoustic phonons

Electron-phonon coupling in semiconductor quantum dots plays a significant role in determining the optical properties of excited excitons, especially the spectral nature of emitted photons. This paper presents a comprehensive theory and analysis of emission spectra from artificial atoms or quantum dots coupled to structured photon reservoirs and acoustic phonons, when excited with incoherent pump fields. As specific examples of structured reservoirs, we chose a Lorentzian cavity and a coupled cavity waveguide, which are of current experimental interest. For the case of optical cavities, we directly compare and contrast the spectra from three distinct theoretical approaches to treat electron-phonon coupling, including a Markovian polaron master equation, a non-Markovian phonon correlation expansion technique and a semiclassical linear susceptibility approach, and we point out the limitations of these models. For the cavity-QED polaron master equation, we give closed form analytical solutions to the phonon-assisted scattering rates in the weak excitation approximation, fully accounting for temperature, cavity-exciton detuning and cavity dot coupling. We show explicitly why the semiclassical linear susceptibility approach fails to correctly account for phonon-mediated cavity feeding. For weakly coupled cavities, we calculate the optical spectra using a more general reservoir approach and explain its differences from the above approaches in the low Q limit of a Lorentzian cavity. We subsequently use this general reservoir to calculate the emission spectra from quantum dots coupled to slow-light photonic crystal waveguides, which demonstrate a number of striking photon-phonon coupling effects. Our quantum theory can be applied to a wide range of photonic structures including photonic molecules and coupled-cavity waveguide systems

Controlled Forgetting: Targeted Stimulation and Dopaminergic Plasticity Modulation for Unsupervised Lifelong Learning in Spiking Neural Networks

Stochastic gradient descent requires that training samples be drawn from a uniformly random distribution of the data. For a deployed system that must learn online from an uncontrolled and unknown environment, the ordering of input samples often fails to meet this criterion, making lifelong learning a difficult challenge. We exploit the locality of the unsupervised Spike Timing Dependent Plasticity (STDP) learning rule to target local representations in a Spiking Neural Network (SNN) to adapt to novel information while protecting essential information in the remainder of the SNN from catastrophic forgetting. In our Controlled Forgetting Networks (CFNs), novel information triggers stimulated firing and heterogeneously modulated plasticity, inspired by biological dopamine signals, to cause rapid and isolated adaptation in the synapses of neurons associated with outlier information. This targeting controls the forgetting process in a way that reduces the degradation of accuracy for older tasks while learning new tasks. Our experimental results on the MNIST dataset validate the capability of CFNs to learn successfully over time from an unknown, changing environment, achieving 95.36% accuracy, which we believe is the best unsupervised accuracy ever achieved by a fixed-size, single-layer SNN on a completely disjoint MNIST dataset